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Abstract 


The  extensive  use  of  local  networks  is  beginning  to  drive 
requirements  for  internetwork  facilities  that  connect  these  local 
networks.  In  particular,  the  availability  of  multicast  addressing  in 
many  local  networks  and  its  use  by  sophisticated  distributed 
applications  motivates  providing  multicast  across  internetworks. 

In  this  paper,  we  propose  a  model  of  service  for  multicast  in  an 
internetwork,  describe  how  this  service  can  be  used,  and  describe 
aspects  of  its  implementation,  including  how  it  would  fit  into  one 
existing  internetwork  architecture,  namely  the  US  Dol)  Internet 
Architecture.1  i 

1.  Introduction 

Multicast  is  the  transmission  of  a  datagram  packet  to  a  set  of 
/.cro  or  more  destination  hosts  in  a  network  or  internetwork,  with 
a  single  address  specifying  the  set  of  destination  hosts.  l;or 
example,  hosts  A.  II.  ('  and  I)  may  be  associated  with  multicast 
address  X.  On  transmission,  a  packet  with  destination  address  X  is 
delivered  with  datagram  reliability  to  hosts  A.  U.  C  and  I). 

Multicast  has  two  primary  uses,  namely  distributed  binding 
and  multi-destination  delivery.  It  is  useful  for  binding  when  one 
or  more  of  a  sc  of  hosts  contain  the  desired  object  but  particular 
host  addresses  arc  not  known,  only  a  multicast  address.  For 
example,  in  a  distributed  file  system,  all  the  file  servers  may  be 
associated  with  one  multicast  address.  To  bind  a  file  name  to  a 
particular  server,  a  client  sends  a  query  packet  containing  the  file 
name  to  the  file  server  multicast  address,  which  is  delivered  to  all 
the  file  servcis.  The  server  that  recognizes  the  file  name  (hen 
responds  to  Inc  client,  allowing  sulisequciit  interaction  directly 
with  that  server  hixl.  Ibis  also  illustrates  the  use  of  multicast  for 
loficul  aiilrcssini}  Ihc  multicast  address  for  a  group  of  hosts  can 
denote  /unction  rather  than  location.  One  can  similarly  associate 
the  grtrup  of  lime  servers,  name  servers,  computation  servers  and 
so  on  each  win  their  own  multicast  address. 

Multi-destination  delivery  is  useful  to  several  applications, 
including: 

•  distributed,  replicated  databases'- 

•  conferencing1. 

•  distrilruled  parallel  compulation,  including  distributed 
gaming4. 


'This  wort  wa.  sponsored  in  part  by  the  Defense  Advanced  Research 
Projects  Agency  under  contract  N00039-8J-K-0-I3I  and  National  SciciM' 
Foundation  Grant  DCK -83-32048. 


Ideally,  multicast  transmission  to  a  set  of  hosts  is  not  more 
complicated  or  expensive  for  the  sender  than  transmission  to  a 
single  host.  Similarly,  multicast  transmission  should  not  be  more 
expensive  for  (he  network  than  traversing  the  shortest  path  tree 
that  connects  the  sending  host  to  the  hosts  identified  by  the 
multicast  address. 

Multicast,  transmission  to  a  set  of  hosts.  Is  properly 
distinguished  from  from  broaJcast.  transmission  to  all  hosts  on  a 
network  or  internetwork.  Broadcast  is  not  a  generally  useful 
facility  since  there  arc  few  reasons  for  communicating  with  all 
hosts.  In  fact,  it  is  host  viewed  as  an  "accident  of  the  technology" 
for  broadcast  networks  in  ihc  same  way  that  self-modifying 
programs  are  an  accident  of  the  technology  for  stored  program 
machines:  just  because  the  technology  provides  it  docs  not  mean 
it  Is  efficient  nr  safe  to  use.  A  proper  multicast  facility  allows 
efficient  transmission  to  multiple  hosts  while  avoiding 
unnecessary  loading  of  the  network  and  receiving  hosts  that  arises 
with  broadcast 

Multicast  Is  now  available  in  standard  local  networks*.  For 
example,  the  Fibcrnc!*’  provides  247  multicast  addresses.  Sending 
a  packet  to  an  Fihcrnct  multicast  address  delivers  it  (with 
datagram  reliability)  to  the  set  of  hosts  listening  to  that  multicast 
address.  A  variety  of  local  network  applications  and  systems 
make  use  of  this  facility.  I  br  instance,  ihc  V  distributed  system7 
uses  network-level  multicast  for  implementing  efficient 
operations  on  groups  of  processes  spanning  multiple  machines. 
Similar  use  is  being  nude  for  replicated  databases'  and  other 
distributed  applications®.  Providing  multicast  in  the  internetwork 
environment  would  allow  porting  such  local  network  distributed 
applications  to  the  internetwork,  as  well  as  nuking  some  existing 
internetwork  applications  more  robust  and  imrtahlc  (by.  for 
example,  removing  wired- in  lists  of  addresses,  such  as  gateway 
addresses). 

In  current  internetwork  environments,  an  application  logically 
requiring  multicast  must  send  individually  addressed  packets  to 
each  recipient  there  arc  two  problems  with  this  approach. 
Firstly.  requiring  the  sending  host  to  know  the  specific  addresses 
of  all  the  recipients  defeats  its  use  as  a  binding  mechanism.  I -or 
example,  a  diskless  workstation  needs  on  bool  to  determine  the 
network  address  of  a  disk  server  and  it  is  undesirable  to  "wire  in" 
specific  network  addresses  With  a  multicast  facility.  Ihc  multicast 
address  of  the  disk  servcis  (or  name  servcis  that  holds  tire  address 
ol  the  disk  server) can  he  n ell  known,  allowing  tltc  workstation  to 
transmit  its  initial  queries  to  this  address. 

Secondly,  transmitting  multiple  copies  of  lire  same  packet 
makes  incliirictil  use  of  net w oik  bandwidth,  gateway  resources 
and  sender  resources  For  instance,  the  stmc  packet  may 
repeatedly  traverse  the  same  network  links  and  pass  through  the 
same  gateways  Furthermore.  Ihc  network  level  cannot  recognize 
mu  It  i-destt  nation  delivery  to  lake  advantage  of  multicast  facilities 
that  the  underlying  netwoik  technologies  may  provide.  For 
example,  local  area  bus.  ling,  or  radio  networks  and  even 
satellite-based  wide-atej  networks  can  provide  efficient  multicast 
deliveiy  directly  Besides  using  excessive  communication 
resources,  the  use  of  multiple  tiaiisnussions  to  effect  multicast 
severely  limits  the  amount  of  parallelism  in  transmission  and 


2.  The  Host  Group  Model 


processing  that  can  be  achieved  compared  to  an  integrated 
multicast  facility. 

In  this  paper,  we  describe  a  model  of  multicast  service  we  call 
host  poups  and  discuss  aspects  of  implementing  this  service  in  a 
datagram  internetwork  environment.  We  argue  that  it  is  feasible 
to  implement  this  facility  in  an  internetwork  as  an  extension  of 
the  existing  “unicast"  internetwork  datagram  model  and 
mechanism. 

We  restrict  ourselves  to  the  communication  environment  of  a 
datagram-based  internetwork,  like  the  II’9  or  XNS10  internetwork 
architectures.  In  these  architectures,  all  hosts  employ  a  common 
internetwork  datagram  formal  and  a  common  internetwork 
addressing  convention  to  identify  the  sources  and  destinations  of 
datagrams.  On  transmission,  an  internetwork  datagram  is 
delivered  to  its  destination  address  with  “best  efforts"  reliability, 
via  the  transmission  services  of  the  underlying  networks  and  (he 
relaying  serv  ices  of  die  gateways  This  service  best  corresponds  to 
OSI  layer  .1  or  the  network  level  in  providing  host-to-host 
delivery.  Reliable  delivery,  including  error  handling  and  Dow 
control,  is  handled  by  higher-level  protocols  that  ojvcralc  in  terms 
of  internetwork  datagrams. 

I  igurc  I  illustrates  a  heterogeneous  collection  of  independent 
networks  interconnected  by  hosts  that  serve  as  slorc-and-forward 
gateways  typical  of  datagram  internet  wot  ks. 


y  SatuHite  Network 
Wide  Area  Network 


mmm  Local  Area  Network 
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figure  I  A  Typical  Internetwork 

In  I  'igurc  I.  a  satellite  network  and  a  wide  area,  slorc-and-forward 
network  connect  several  local  area  networks  as  well  as  individual 
hosis  the  combination  of  broadcast  and  point-to-point 
technology  plus  the  usual  coiuplicalious  ot  ilHTciem  speeds.  (|c|;ly 
and  nuvumint  transmission  unit  make  an  efficient 
impleiiienlalion  of  mull  n  asi  a  challenge. 

Ihe  ncvi  sect Htu  describes  the  host  group  model  of  multicast 
scoicc  Section  .1  describes  the  implementation  strategy  we 
promise  Section  4  describes  how  tins  extension  fits  into  the 
current  US  IX>I)  Internet  architecture  and  briefly  touches  on 
other  internetwork  architectures  Section  5  illustrates  how  this 
facility  can  be  used  by  a  variety  of  applicativins  Section  6  relates 
this  model  lo  other  proposals,  finally,  we  conclude  with  remarks 
on  the  status  of  our  ex ikriniciit.nl  prototype  implementation  of 
host  gioups  and  our  future  directions  for  investigation. 


In  an  internetwork  designed  in  the  host  group  model,  each 
internetwork  address  identifies  a  host  group.  A  host  group  is  a  set 
of  zero  or  more  hosts  in  one  internetwork.3  When  an 
internetwork  packet  is  sent,  it  is  delivered  with  “best  efforts” 
datagram  reliability  to  all  members  of  the  host  group  identified 
by  the  internetwork  address  in  the  packet  destination  field. 

I  he  sender  need  not  be  a  member  of  the  destination  group. 
We  refer  to  such  a  group  as  open,  in  contrast  to  a  closed  group 
where  only  members  arc  allowed  to  send  to  the  group.  We  chose 
to  provide  open  groups  because  they  arc  more  flexible  and  more 
consistent  as  an  extension  of  conventional  unicasts  models  (even 
though  they  arc  harder  to  implement). 

Dynamic  management  of  group  membership  provides  flexible 
binding  of  internetwork  addresses  to  hosts.  I  lusts  may  join  and 
leave  grixips  over  lime.  A  host  may  also  belong  to  more  than  one 
group  at  a  time,  finally,  a  host  may  belong  lo  no  groups  at  times, 
during  which  that  host  is  unreachable  within  the  internetwork 
architecture.  In  fact,  an  internetwork  Inst  need  not  have  an 
individual  internetwork  address  at  all.  Some  hosts  may  only  be 
associated  with  multi-host  group  addresses,  for  instance,  there 
may  be  no  reason  to  contact  an  individual  lime  server  in  the 
internetwork,  so  time  scrvcis  would  not  require  individual 
addresses.  Similarly,  a  hank  of  shared  processors  may  he  identical 
from  the  standpoint  of  clients  and  only  acquire  individual 
intcrnclwork  addresses  while  they  arc  servii  g  individual  clients. 

Internetwork  addresses  are  dynamically  allocated  for  transient 
groups,  groups  that  often  last  only  as  long  as  the  execution  of  a 
single  distributed  program  A  range  of  host  group  identifiers  is 
reserved  for  identify  ing  permanent  groups  One  use  of  permanent 
host  groups  identifiers  is  for  host  groups  with  standard  logical 
meanings  such  as  “name  server  group".  "Ikxil  server  group”, 
"internetwork  monitor  group",  etc.  I’crmancntly  assigned 
addresses  arc  also  used  for  conventional  single-host  addresses. 

Ihe  host  group  model  of  internetwork  generalizes  the  binding 
of  internetwork  addresses  to  internetwork  hosts  by  allowing  one 
address  lo  hind  to  multiple  hosts  on  multiple  networks,  more  than 
one  address  lo  be  bound  (in  part)  to  one  host,  and  the  binding  of 
an  address  to  bust  lo  lie  dynamic,  i.c.  possible  lo  modify  under 
application  control,  for  performance  reasons,  the  conventional 
ease  of  single-member  groups  is  handled  specially  as  an 
optimization.  A  range  of  internetwork  addicsscs  are  reserved  for 
designating  groups  of  at  most  one  inlet  network  host,  allowing  the 
delivery  nk-chanism  to  make  appropriate  optimizations. 
Moreover,  if  the  internetwork  address  is  statically  bound  lo  a  host 
pemiancnlly  attached  through  one  network,  a  network  identifier 
can  be  embedded  as  a  subfield  of  its  internetwork  address  in 
order  to  simplify  gateway  routing.  As  should  be  ap|varcnt.  this 
special  case  corresponds  to  the  unicast  facility  provided  by  several 
current  datagram-based  internetwork  architectures,  including  IP 
and  XNS.  thus.  Ihe  host  group  model  is  a  compatible  extension 
of  these  architectures. 

'Ihe  following  subsections  provide  further  details  of  the  model. 

2.1  Host  Group  Management 

Dynamic  binding  of  internetwork  addresses  to  hosts  is 
managed  by  the  following  three  otierations  available  to  higher- 
level  protocols  or  applications:* 


3ln  reality,  the  internetwork  address  is  bound  to  network  interfaces  or 
host  access  ports,  not  Uie  host  machine  per  se. 

*ln  this  procedure  call  notation,  the  arguments  for  an  operation  nro 
listed  in  parentheses  alter  the  operation  name,  and  the  returned  values,  if 
any,  are  listed  alter  a  -->  symbol. 


CreateGroup  (  type  ) 

— >  outcome,  group-address,  access-key 

requests  the  creation  of  a  new  transient  host  group  with  the 
invoking  host  as  its  only  member.  The  type  argument  specifies 
cither  a  general  group  or  a  onc-membcr-only  group  plus  whether 
the  group  is  restricted  or  unrestricted.  A  restricted  group  restricts 
membership  based  on  the  access-key.  Only  hosts  presenting  a 
valid  host  access  key  arc  allowed  to  join.  AH  unrestricted  host 
groups  have  a  null  access-key.  outcome  indicates  whether  the 
request  is  approved  or  denied.  If  it  is  approved,  a  new  transient 
group  address  is  returned  in  group-address,  access-key  is 
the  protection  key  (or  password)  associated  with  the  new  group. 
Ibis  should  fail  only  if  there  arc  no  free  transient  group 
addresses. 

JoinGroup  (  group-address,  access-key  ) 
-->  outcome 

requests  that  the  invoking  host  become  a  member  of  the 
identified  host  group  (permanent  or  transient),  outcome 
indicates  whether  the  request  is  approved  or  denied.  A  request 
may  be  denied  if  the  access  key  is  invalid. 

LeaveGroup  (  group-address  ) 

-->  outcome 

requests  that  the  invoking  host  be  drop|>cd  from  membership  in 
the  identified  croup  (permanent  or  transient).  outcome 
indicates  whether  the  request  is  approved  or  denied. 

there  is  no  operation  to  destroy  a  transient  hast  group  because 
a  transient  host  group  is  deemed  to  no  longer  exist  when  its 
membership  gres  to  /cro. 

Note  that  in  conventional  internetworks  allocation  and 
binding  of  internetwork  addresses  is  typically  performed  statically 
by  internetwork  administrators 

2.2  Packet  Transmission 

Transmission  of  a  packet  in  the  host  group  model  is  controlled 
by  two  parameters  of  scope,  one  being  the  destination 
internetwork  address  and  the  other  being  the  "distance"  to  the 
members  in  live  group.  In  particular, 

Send  (  dest-address ,  source-address, 

data,  distance  ) 

transmits  the  specified  data  in  an  internetwork  datagram  to  the 
hosts  in  the  host  group  specified  by  dest-address  that  arc 
within  the  specified  distance  Ihc  destination  address  is  thus 
similar  to  conventional  networks  except  that  delivery  may  be  to 
multiple  hosts:  ihc  distance  parameter  requires  further  discussion 

Distance  may  Ik  measured  in  several  ways,  including  number 
of  network  hops,  lime  to  deliver  and  what  might  Ik  called 
administrative  distance.  Administrative  distance  refcis  to  the 
distance  between  (he  administrations  of  two  different  networks. 
Tor  example,  in  a  company  the  networks  of  the  research  group 
aim  advanced  development  group  might  be  considered  quite 
dose  to  each  other,  networks  of  the  corporate  management  more 
distant,  and  networks  of  other  companies  much  more  distant. 
One  may  wish  to  restrict  a  query  to  members  within  one's  own 
administrative  domain  because  servers  outside  that  domain  may 
not  be  trusted.  Similarly,  error  reporting  outside  of  an 
administrative  domain  may  not  be  productive  and  may  in  fact  be 
confusing. 

Resides  limiting  the  scope  of  transmission,  the  distance 
parameter  cun  be  used  to  control  Ihc  scope  of  multicast  as  a 


binding  mechanism  and  to  implement  an  expanding  scope  of 
search  for  a  desired  service.  Tor  instance,  to  locale  a  name  server 
familiar  with  a  given  name,  one  might  check  with  nearby  name 
servers  and  expand  the  distance  (by  incrementing  Uic  distance  on 
retransmission)  to  include  more  distant  name  servers  until  the 
name  is  found. 

To  reach  all  members  of  a  group,  a  sender  specifics  the 
maximum  value  for  the  distance  parameter.  This  maximum  must 
exceed  the  "diameter"  of  the  internetwork. 

The  distance  parameter  can  be  viewed  as  an  extension  of  the 
time-to-live  or  hop  count  parameters  that  arc  used  in  several 
internetwork  architectures  to  prevent  infinite  routing  cycles.  In 
those  cases,  live  distance  parameter  basically  ensures  that  the 
delivery  mechanism  only  expends  a  finite  amount  of  work  in 
delivery  and  therefore  discards  a  packet  caught  in  a  tooling  loop. 
'The  distance  parameter  in  the  host  group  model  refines  this  finite 
bound  into  further  gradations. 

Rather  than  define  specific  semantics  of  the  distance 
parameter  in  the  model,  we  sec  it  having  a  refinement  of  the 
semantics  of  Ihc  timc-lo-iivc  or  hop  count  parameters  specific  to 
each  internetwork  architecture.  However,  in  all  cases,  there  is  a 
need  for  well-known  boundaries  values  that  coincide  with 
administrative  domains.  Tor  instance,  there  is  a  need  for  a 
distance  value  that  corresponds  to  "not  outside  this  local 
network". 

Packet  reception  is  the  same  as  conventional  architectures. 
That  is. 

Receive  () 

-->  dest-address,  source-address,  data 

returns  ihc  next  internetwork  datagram  that  is.  or  has  been, 
received. 

2.3  Delivery  Requirements 

We  identify  several  requirements  for  the  packet  delivery 
mechanism  that  are  essential  to  host  groups  being  a  useful  and 
used  facility. 

firstly,  given  the  predominance  of  broadcast  local-area 
networks  and  the  locality  of  communication  to  individual 
networks,  the  delivery  mechanism  must  be  able  to  exploit  the 
hardware's  capability  for  very  efficient  multicast  within  a  single 
local-area  network. 

Secondly.  Ihc  delivery  mechanism  must  scale  in  sophistication 
to  efficient  delivery  across  the  internetwork  as  internetworks 
acquire  high-s|Kcd  wide-area  communication  links  and  high 
performance  gateways.  Ihc  former  arc  being  provided  by  Ihc 
introduction  of  high-speed  satellite  channels  and  long-haul  fiber 
optic  links.  Ihc  latter  are  made  feasible  by  the  falling  cost  of 
nKmory  and  processing  |>owcr  plus  the  increasing  importance  in 
controlling  access  to  relatively  unprotected  local  network 
environments  A  host  group  delivery  mechanism  must  be  able  to 
take  advantage  of  these  trends  as  they  materialize. 

finally,  the  delivery  mechanism  most  avoid  "systematic 
errors"  in  delivery  lo  members  of  the  host  group  that  is.  a  small 
number  of  repeated  transmissions  must  result  in  delivery  to  all 
group  members  within  the  specified  distance,  unless  a  iiKinbcr  is 
disconnected  or  has  failed  We  refer  to  this  properly  as  covcmge. 
In  general,  most  reliable  protocols  make  this  basic  assumption  for 
unicast  delivery.  It  ts  important  to  guarantee  Ihis  assumption  for 
multicast  ns  well  or  else  applications  using  multicast  may  fait  in 
unex|KCled  wavs  when  coverage  is  not  provided  l  or  efficiency, 
the  multicast  delivery  mechanism  should  also  avoid  regularly 
delivering  multiple  copies  of  a  packet  to  individual  hosts. 

failure  notification  is  not  viewed  as  an  essential  requirement 
given  Ihc  datagram  semantics  of  delivery  I  lowcver,  a  host  group 
extension  of  internetwork  architectures  such  as  IP  and  XNS 


should  preside  "hinf-lcvcl  failure  nolificalion  as  (he  natural 
extension  of  their  failure  notification  for  unicast 

3.  Implementation 

In  this  section,  we  sketch  a  design  for  implementing  (lie  host 
group  model  in  a  datagram  internetwork.  This  description  of  the 
design  is  given  to  further  support  the  feasibility  of  the  host  group 
model  as  well  as  point  out  some  of  (he  problems  yet  to  be 
addressed. 

Implementation  of  host  groups  involves  implementing  a 
binding  mechanism  (binding  internetwork  addresses  to  zero  or 
more  hosts)  and  a  packet  delivery  mechanism  (delivering  a  packet 
to  each  host  to  which  its  destination  address  binds).  This  facility 
tils  most  naturally  into  the  gateways  of  the  internetwork  and  the 
switching  nodes  of  the  constituent  poim-lo-|H>inl  networks  (as 
opposed  to  separate  machines)  because  multicast  binding  and 
delivery  is  a  natural  extension  of  the  unicast  binding  and  delivery 
(i.c.  muting  plusstorc-and-forward).  'Ihal  is.  a  multicast  packet  is 
routed  and  transmitted  to  multiple  destinations,  rather  than  to  a 
single  destination. 

A  gateway  in  a  host  group  internetwork  Ls  thus  viewed  ns  a 
"communication  server",  providing  multicast  delivery  and  host 
group  management.  Ihc  multicast  delivery  service  is  invoked 
implicitly  by  sending  packets  addressed  to  host  groups,  with 
unicast  delivery  as  a  special  case.  Ihc  group  management  service 
is  invoked  explicitly  using  a  request-response  transaction  protocol 
between  ihc  client  hosts  and  the  server  gateways.  In  addition  to 
die  operations  for  creating  transient  host  groups  and  adding  and 
deleting  host  memberships  in  groups  (Section  2.1).  the  gateway 
supports  operations  for  administrative  allocation  of  permanent 
group  addresses,  including  static,  single  host  group  addresses  (i.c. 
unicast  addresses). 

In  the  following  description,  we  start  with  a  basic,  simple 
implementation  dial  provides  coverage  and  (hen  refine  this 
mechanism  with  various  optimizations  to  improve  efficiency  of 
delivery  and  group  management. 

3.1  Basic  Implementation 

A  hast  group  defines  a  network  yroup.  which  is  the  set  of 
networks  containing  current  members  of  the  host  group.  When  a 
packet  is  sent  to  a  host  group,  a  copy  is  delivered  to  each  network 
in  the  corresponding  network  group.  'then.  within  each  network, 
a  copy  is  delivered  lo  each  host  belonging  to  (lie  group 

To  support  such  multicast  delivery,  every  internet  gateway 
maintains  the  following  data  structures: 

•  routing  table:  conventional  internetwork  routing 
information,  including  the  distance  and  direction  to  the 
nearest  gateway  on  every  network. 

•  network  membership  table:  A  set  of  records,  one  for  every 
currently  existing  host  group  Ihe  network  membership 
record  for  a  group  lists  the  network  group,  i.c.  the  networks 
that  contain  members  of  (he  group. 

•  local  host  membership  table:  A  set  of  records,  one  for  each 
host  group  ihal  has  members  on  directly  attached  networks. 
Inch  It  mil  host  membership  record  indicates  Ihc  local  hosts 
that  arc  members  of  the  associated  host  group  l  or 
networks  that  support  multicast  or  broadcast.  Ihc  record 
may  contain  only  the  local  network- spec t/ic  multicast 
address  used  by  ihc  group  plus  a  count  of  local  members. 
Otherwise,  local  group  members  may  be  identified  by  a  list 
of  unicast  addresses  to  be  used  in  the  software 
implementation  of  multicast  within  Ihc  network. 

A  host  invokes  ihc  multicast  delivery  service  by  sending  an 
internetwork  datagram  loan  immediate  neighbour  gateway  (i.c.  a 
gateway  that  is  directly  attached  lo  Ihc  same  network  tis  the 
sending  host)  Upon  receiving  a  datagram  from  a  directly 
attached  network,  a  gateway  looks  up  the  network  membership 


record  corresponding  to  the  destination  address  of  the  datagram. 
For  each  of  the  networks  listed  in  die  membership  record,  the 
gateway  consults  its  routing  (able.  If.  according  to  the  routing 
table,  a  member  network  Ls  directly  attached,  the  gateway 
transmits  a  copy  of  the  datagram  on  that  network,  using  the 
network -s|>ccific  multicast  address  allocated  for  the  group  on  that 
network.  For  a  member  nciwork  that  is  not  directly  attached  and 
Ls  within  the  distance  constraint  specified  in  Ihc  datagram,  the 
gateway  creates  a  copy  of  the  datagram  with  an  additional  inter- 
gateway  header  identifying  die  destination  network.  This  inter- 
gateway  datagram  is  forwarded  to  (lie  nearest  gateway  on  the 
destination  network,  using  conventional  storc-and-forward 
routing  techniques.  At  the  gateway  on  Ihc  destination  network, 
Ihc  datagram  Ls  stripped  of  its  inter-gateway  header  and 
transmitted  lo  the  group's  multicast  address  on  that  network. 
Member  networks  that  are  beyond  the  datagram's  distance 
constraint  arc  ignored. 

Ihc  network  membership  records  and  the  net  work -specific 
multicast  structures  arc  updated  in  response  to  group 
management  requests  from  hosts.  A  host  sends  a  request  to 
create,  join,  or  leave  a  group  to  an  immediate  neighbour  gateway. 
If  the  host  requests  creation  of  a  group,  a  new  nciwork 
membership  record  is  created  by  the  serving  gateway  and 
distributed  to  all  other  gateways.  If  the  host  is  the  first  on  its 
nciwork  to  join  a  group,  or  if  the  host  is  the  last  on  its  network  to 
leave  a  grixip.  the  group's  network  membership  record  is  updated 
in  all  gateways.  Ihc  updates  need  not  be  performed  atomically  at 
all  gateways,  due  to  Ihe  datagram  delivery  semantics:  hosts  can 
tolerate  m isrouted  and  lost  packets  caused  by  temporary  gateway 
inconsistencies,  as  long  as  the  inconsistencies  arc  resolved  within 
normal  host  retransmission  periods  In  this  respect,  the  network 
membership  data  is  similar  to  Ihc  network  reachability  data 
maintained  by  conventional  muting  algorithms,  and  can  be 
handled  by  similar  mechanisms. 

In  many  eases,  a  host  joins  a  group  Ihal  already  has  members 
on  the  some  network,  or  leaves  a  group  that  has  remaining 
members  on  the  same  network.  This  is  then  a  local  matter 
between  Ihc  hnsts  and  gateways  on  n  single  network:  only  the 
local  host  membership  table  needs  to  be  updated  to  include  or 
exclude  the  host: 

Mils  basic  implementation  strategy  meets  the  delivery 
requirements  staled  at  ihe  end  of  Section  ?  However,  it  is  far 
fiom  optimal,  in  terms  of  either  delivery  efficiency  or  group 
management  overhead  One  simple  improvement  is  to  recognize 
Ihe  important  special  ease  of  static,  cme-mcmber-only  groups 
Ihis  again  correqximls  to  the  conventional  unicast  provided  in 
(for  example)  II’  and  XNS.  In  this  case,  the  internetwork  address 
for  the  single-host  group  encodes  within  it  the  network  of  live  one 
host  so  there  is  no  need  to  maintain  a  separate  group  membership 
record  for  Ihal  group  Consequently  ,  the  number  of  group 
membership  records  in  the  gateways  is  greatly  reduced.  Also, 
delivery  to  these  groups  degenerates  to  convent  ional  unicast 
techniques  such  as  currently  used  in  II’  and  XNS 
implementations.  Ilclow.  we  discuss  some  further  refinements  lo 
the  basic  implementation. 

3.2  Multicast  Routing  Between  Networks 

Multicast  routing  among  the  internetwork  gateways  is  similar 
to  storc-and-forward  routing  in  a  point-to-point  network.  Ihc 
mam  difference  is  that  the  link-  between  ihe  nodes  (gateways)  can 
be  a  mixture  of  broadcast  and  unicast-type  networks  with  widely 
different  throughput  and  delay  characteristics.  In  addition, 
packets  arc  addressed  to  networks  rather  titan  hosts  (at  the 
gateway  level). 

We  use  Ihc  extended  reverse  path  forwarding  algorithm  of 
Dalai  and  Metcalfe*'  Although  originally  designed  for 
broadcast,  it  is  a  simple  and  efficient  technique  Ihal  can  serve  well 
for  multicast  delivery  if  network  membership  tecords  in  each 
gateway  are  augmented  with  information  from  neighbouring 


gateways.  'Ihis  algorithm  uses  (he  source  network  identifier, 
rather  than  a  destination  network  identifier  to  make  routing 
decisions.  Since  the  source  address  of  a  datagram  is  a  general 
group  address,  it  cannot  be  used  to  identify  the  source  network  of 
the  datagram:  the  first  gateway  must  add  a  header  specifying  the 
source  network  This  approach  minimizes  redundant 
transmissions  when  multiple  destination  networks  arc  reachable 
across  a  common  intergateway  link,  a  problem  with  the  basic 
implementation  described  earlier. 

Note  that  we  eliminated  from  consideration  techniques  that 
fail  to  deliver  along  the  branches  of  the  shortest  delay  tree  rooted 
at  the  source,  such  as  Wall's  center-based  forwarding’2  because 
this  compromises  the  meaning  of  the  multicast  distance  parameter 
and  detracts  from  multicast  performance  in  general.  We  also 
rejected  (he  approach  of  having  a  multicast  packet  carry  more 
than  one  network  identifier  in  its  inter-gateway  header  to  indicate 
multiple  destination  networks  because  the  resulting  variable 
length  headers  would  cause  buffering  and  fragmentation 
problems  in  the  gateways. 

3.3  Multicasting  Within  Networks 

A  simple  optimi/ation  within  a  network  is  to  have  the  sender 
use  the  local  multicast  address  of  a  host  group  for  its  initial 
transmission  Ihis  allows  the  local  host  group  members  to  receive 
the  transmission  immediately  along  with  the  gateways  (which 
must  now  "eavesdrop"  on  all  multicast  transmissions).  A  gateway 
only  forwards  the  datagram  if  the  destination  host  group  includes 
members  on  other  networks.  This  scheme  reduces  the  cost  to 
reach  local  group  members  to  one  packet  transmission  from  two 
required  in  the  basic  implementation5  so  transmission  to  local 
members  is  basii-alty  as  efficient  as  the  local  multicist  support 
provided  by  the  network. 

A  similar  opportunity  for  reducing  packet  traffic  arises  when  a 
datagram  nuts  traverse  a  network  to  get  from  one  gateway  to 
another,  and  that  network  also  holds  members  of  the  destination 
group  Again,  use  of  a  network-specific  multicast  address  which 
includes  mem  tier  hosts  plus  gateways  can  achieve  the  desired 
effect.  I  lowcver.  In  ihis  ease,  hosts  must  lie  prepared  to  accept 
datagrams  that  include  an  inter-gateway  header  or.  alternatively, 
every  datagram  must  include  a  spare  field  in  its  header  for  use  by 
gateways  in  lieu  of  an  additional  inter-gateway  header. 

3.4  Distributing  Membership  Information 

A  refinement  to  host  group  membership  maintenance  is  to 
store  the  host  group  membership  record  for  a  group  only  in  those 
gateways  that  are  directly  connected  to  member  networks. 
Information  about  other  groups  is  cached  in  the  gateway  only 
while  it  is  required  to  route  to  those  other  groups  When  a  gateway 
receives  a  datagram  to  be  forwarded  to  a  group  fur  which  it  has 
no  network  membership  record  (which  can  only  happen  if  the 
gateway  is  not  directly  connected  to  a  member  network),  it  takes 
the  following  action,  the  gateway  assumes  temporarily  that  the 
destination  group  has  members  on  even  network  in  the 
intei  net  work,  except  those  directly  attached  to  the  sending 
gateway,  and  routes  the  datagram  accordingly.  In  the  inter¬ 
gateway  header  of  the  outgoing  packet,  the  gateway  sets  a  bit 
indicating  that  it  wishes  to  receive  a  copy  of  the  network 
membership  record  for  the  destination  host  group.  When  such  a 
datagram  reaches  a  gateway  on  a  nKinber  network,  that  gateway 
sends  a  copy  of  the  mcmlvcrship  record  liack  to  the  requesting 
gateway  and  dears  the  copy  request  bit  in  the  datagram. 

Copies  of  network  membership  records  sent  to  gateways 
outside  of  a  group's  member  networks  arc  etched  for  use  m 
subsequent  transmissions  by  those  gateways.  Iliat  raises  the 


5Onc  unicast  transmission  from  sender  to  gateway  and  one  mtillk.Ht 
transmission  from  gateway  to  local  group  membcri 


danger  of  a  stale  cache  entry  leading  to  systematic  delivery 
failures.  To  counter  that  problem,  the  inter-gateway  header 
contains  a  field  which  is  a  hash  value  or  checksum  on  the  network 
membership  record  used  to  route  the  datagram.  Gateways  on 
member  networks  compare  the  checksum  on  incoming  datagrams 
with  their  up-to-date  records.  If  the  checksums  don't  match,  an 
up-to-date  copy  of  the  record  is  relumed  to  the  gateway  with  the 
bad  record. 

'Ihis  caching  strategy  minimizes  intergateway  traffic  for  groups 
that  are  only  used  within  one  network  or  within  the  set  of 
networks  on  which  members  reside,  die  expected  common  eases. 
Partial  replication  with  caching  also  reduces  the  overhead  for 
network  traffic  to  disseminate  updates  and  keep  all  copies 
consistent.  Finally,  it  also  reduces  the  space  cost  for  data  in  large 
internetworks  with  targe  numbers  of  multiple  host  groups. 

We  have  nut  addressed  here  the  problem  of  maintaining 
up-to-date,  consistent  network  membership  records  within  the  set 
of  gateways  connected  to  members  of  a  group.  Ihis  can  be 
viewed  as  a  distributed  database  problem  which  has  been  well 
studied  in  other  contexts.  The  loose  consistency  requirements  on 
network  membership  rccoids  suggest  that  the  techniques  used  in 
Grapevine1-’  might  be  useful  for  this  application. 

4.  Integration  into  the  DoD  Internet 

To  show  how  the  host  group  model  can  be  supponed  by 
straightforward  extension  of  an  existing  internetwork 
architecture,  we  outline  how  it  might  fit  into  Ihc  US  l)ol) 
Internet 

Ihc  current  Internet  provides  unicast  datagram  delivery 
between  hosts  on  a  wide  variety  of  networks,  both  local-area  and 
wide-area,  broadcast  and  point-to-point.  An  Internet  address  is  a 
J.’-bit  value  consisting  of  two  subficlds:  a  network  number  and  a 
host-wilhin-nciwork  number  l-vcry  Internet  gateway  maintains  a 
routing  table  that  specifics  Ihc  distance  and  direction  to  every 
network  in  the  Internet,  relative  to  the  gateway.  Ihus.  given  a 
datagram,  a  gateway  ran  determine  from  die  network  number 
sublicld  of  its  destination  address,  where  to  send  it  next  on  the 
path  towards  its  destination.  When  the  datagram  reaches  a 
gateway  into  its  destination  network,  that  gateway  maps  the 
hosl-withiii-nclwork  number  to  a  local  network  address  for  final 
delivery. 

the  existing  architecture  supports  out  model  of  static,  onc- 
mcmhcr-only  groups  We  extend  Ihis  architecture  to  support 
multiple  host  groups  In  reserving  a  single  network  numlKr  to 
identify  all  such  groups.  latch  multiple  bust  group  is 
distinguished  by  a  unique  value  in  llic  hosl-within-nctwork 
subfield  of  its  internet  address  Ihc  Intcrnel  gateways  arc 
augmented  witli  the  data  structures  and  procedures  discussed  in 
Section  .1  to  support  internet  multicast 

An  II*  datagram  contains  a  "time  to  live"  field  which  is 
decremented  by  the  gateways  once  a  second  and  on  every 
network  hop  If  the  lime  to  live  goes  to  zero  before  Ihc  datagram 
reaches  ns  destination,  the  datagram  is  discaidcd.  In  the  host 
group  implementation,  this  field  is  used  to  limit  Ihc  delivery 
distance  of  multicasts. 

Ollier  datagram  internetwork  architectures  yield  to  similar 
extensions.  l-'or  example,  the  Xerox  Network  Systems 
architecture  is  essentially  identical  to  the  l)ol)  Internet  with 
regards  to  address  encoding  (  wtwork.  host -wilhin-nctwork)  and 
contents  of  routing  tables  XNS  datagrams  contain  a  hop  count 
field  dial  can  be  used  for  multicast  scope  control. 

Ihc  proposed  ISO  internetwork  protocol1,1  provides  the  same 
style  of  internetwork  dalagiam  service  as  II’  or  XNS.  Ihc  draft 
proposal  loi  ISO  internetwork  addresses1'  specifics  a  much  more 
complex  strucHiie  than  the  fixed-length,  two  level  hierarchical 
addresses  of  II*  and  XNS  A  more  sophisticated,  possibly 
hierarchical,  distribution  of  the  network  membership  records 
would  Ik  appropriate  for  the  enormous  |Hitential  size  of  the  ISO 


“world  network". 


6.  Related  Work 


5.  Use  of  Multicast 

A  number  of  applications  that  can  use  multicast  have  been 
cited  earlier  in  the  paper,  including  distributed  databases, 
conferencing,  distributed  computation  and  locating  internetwork 
services  Rather  than  describe  these  applications  in  greater  detail, 
we  focus  on  some  general  issues  that  were  identified  in  previous 
work7.  (  This  work  dealt  with  the  use  of  local  network  multicast  in 
a  distributed  operating  system  to  support  the  concept  of 
interprocess  group  communication  where  process  groups  are 
distributed  across  host  groups.) 

A  key  issue  is  providing  reliable  communication  as  required 
by  the  application  firstly,  some  applications,  such  as  real-tune 
conferencing,  do  not  need  reliable  delivery,  assuming  the  periodic 
updates  are  generally  received.  Secondly,  binding  applications, 
such  as  locating  a  name  server,  do  not  require  delivery  to  all  but 
simply  a  positive  response  from  at  least  one  host.  Retransmission 
with  possibly  expanding  scope  of  search  until  a  response  is 
received  provides  the  required  semantics. 

As  an  aside,  one  might  argue  that  the  binding  use  is  only  really 
required  to  locate  a  name  server.  While  true  in  theory,  it  ntay  be 
simpler  for  some  applications  to  locate  other  servers  directly  using 
this  simple  search  protocol  Then  they  do  not  need  to  implement 
the  protocol  to  lookup  a  name  in  the  name  server  as  well  as  this 
simple  search  protocol  to  locate  the  name  server  in  the  first  place, 
l-'or  example,  the  I’ROM  network  loader  for  diskless  workstations 
might  be  simpler  if  it  can  locale  a  boot  server  using  a  bool  server 
group  address  directly  rather  than  going  through  a  name  server. 

I  or  applications  requiring  reliable  delivery,  there  arc  basically 
two  approaches.  'Hie  most  common  approach  is  to  place  die  onus 
for  reliable  delivery  on  the  sender.  Here,  the  sender  knows  the 
membership  of  a  group  and  retransmits  to  the  group  until  it  has 
received  acknowledgements  from  each  group  member.  As  an 
optimization,  the  sender  can  use  unicast  to  retransmit  to 
particular  group  members  if  the  number  of  missing 
acknowledgements  is  relatively  small  compared  to  the  cardinality 
of  the  host  group. 

Ibc  second  approach  places  the  onus  on  the  receivers  to 
implement  reliable  delivery,  what  we  call  publishing.  Il  is  so 
named  because  il  mimics  real  world  publishing  'Ihal  is. 
information  to  be  sent  to  a  group,  the  subscribers,  is  filtered 
though  the  publisher,  which  collates  and  mimlicrs  the  information 
before  issuing  it  to  the  subscribers.  A  suliscribcr  noticing  a 
missing  issue  by  a  gap  in  the  issue  numbers  or  a  new  issue  not 
being  received  in  the  expected  lime  interval  requests  the  back 
issue  from  the  publisher.  Ihus,  instead  of  automatic 
retransmission  until  the  receiver  acknowledges  the  message,  the 
receiver  must  request  retransmission  if  il  is  required. 

A  family  of  reliable  multicast  protocols  is  specified  by  (bang 
and  Maxcmchuk'6  that  combines  both  techniques  built  on  lop  of 
an  unreliable  broadcast  or  multicast  network.  They  describe  a 
protocol  that  guarantees  not  only  that  all  group  members  receive 
all  messages,  but  also  that  they  all  receive  the  messages  in  the 
same  older,  regardless  of  the  mimlier  of  senders  furthermore, 
this  strong  level  of  reliability  is  achieved  with  only  one 
acknowledgement  per  message  in  the  normal  ease,  no  single  point 
of  failure,  and  survival  in  the  face  of  multiple  host  failures  and 
recoveries.  In  another  paper*,  (bang  describes  the  use  of  this 
protocol  to  support  a  distributed,  replicated  database. 

In  general,  the  problem  is  not  implementing  reliable  delivery 
for  multicast  delivery  but  chousing  the  right  trade-off  between 
cost,  performance  and  reliability  as  required  by  the  application 
We  have  briefly  described  some  basic  techniques.  However, 
further  study  is  required  to  understand  these  tradc-olTs  with 
various  applications  and  internetworking  parameters. 


There  is  relatively  little  published  work  on  the  use  or 
implementation  of  internetwork  multicasting. 

Wall's  diesis*2  presents  several  mechanisms  for  ‘performing 
efficient  broadcast  and  multicast  delivery  in  point-to-point 
networks.  Ills  results  can  be  applied  to  providing  multicast 
within  point-to-point  networks  dial  arc  constituents  of  an 
internetwork,  and  to  the  problems  of  multicast  routing  to 
“network  groups”  of  gateways. 

Boggs,  in  his  thesis®,  describes  a  number  of  distributed 
applications  that  arc  impossible  or  very  awkward  to  support 
without  the  flexible  binding  nature  of  broadcast  addressing. 
Although  he  recognizes  that  almost  all  of  his  applications  would 
be  best  served  by  a  multicast  mechanism,  he  advocates  the  use  of 
“directed  broadcast"  because  il  is  easy  to  implement  within  many 
kinds  of  networks  and  can  be  extended  across  an  internetwork, 
without  placing  any  new  burden  on  internetwork  gateways. 
Unfortunately,  broadcasting  has  the  undesirable  side  effect  of 
delivering  packets  to  more  hosts  than  necessary.  Ihus  incurring 
overhead  on  uninvolvcd  parties  and  possibly  creating  security 
problems.  I'unhcrmorc.  directed  broadcasting  supports  simple 
communication  with  unknown  destinations  on  directly  connected 
networks  only:  for  destinations  on  more  distant  networks,  the 
sender  must  know  their  network  numbers  or  |icrform  a  search 
using  gateway  routing  tables. 

Recent  proposals  by  Mogul*7  and  Aguilar’8  have  addressed 
the  issue  of  multi-destination  delivery  within  the  Do!)  Internet. 
Mogul  proj  wises  an  implementation  of  Hogg's  directed  broadcast 
facility.  Aguilar  suggests  allowing  an  IP  datagram  to  carry 
additional  destination  addresses,  which  arc  used  by  the  gateways 
to  route  the  datagram  to  each  recipient  Such  a  facility  would 
alleviate  some  of  the  inefficiencies  of  sending  individual 
datagrams  to  a  group,  but  it  would  not  be  able  to  take  advantage 
of  load  network  multicast  facilities.  More  seriously.  Aguilar's 
scheme  requires  the  sender  to  know  the  individual  IP  addresses  of 
all  members  of  the  destination  group  and  thus  lacks  the  flexible 
binding  nature  of  true  multicast  or  broadcast 

Itlauslcin  ct  at19  discuss  a  variety  of  protocols  for  reliable 
multicast  delivery  based  on  various  (iulcr)network  characteristics 
(e  g.  poiiil-lo-poinl  or  broadcast  or  liotli.  dusters  of  fast  networks 
joined  by  slower  networks,  degree  «>r  multicast  support  provided 
bv  the  networks,  etc )  Ax  well  as  making  a  case  lor  unreliable 
mullicasl  services  at  the  internetwork  level,  their  work  suggests 
wavs  of  achieving  efficient  multicast  among  gateways  in  a 
heterogeneous  inlet  network. 

7.  Concluding  Remarks 

We  have  described  a  model  of  multicast  communication  for 
datagram- based  internetworks.  As  an  extension  of  existing 
internetwork  architcdures.  it  views  unicast  communication  and 
limc-lo-livc  constraints  as  special  eases  of  the  more  general  form 
of  communication  arising  with  mullicasl.  We  have  argued  that 
tilts  model  is  implcmcntable  in  current  and  future  internetworks 
and  that  it  provides  a  powerful  facility  for  a  variety  of 
applications  In  some  cases,  it  provides  a  facility  that  is  required 
for  certain  applications  to  work  in  the  internetwork  environment. 
In  other  eases,  it  provides  a  more  efficient,  robust  and  pnedbly 
more  elegant  way  of  implementing  existing  internetwork 
applications. 

We  arc  currently  implementing  a  prototype  host  group  facility 
as  an  extension  of  IP.  I'or  practical  reasons.  Ihis  prototype 
implements  all  group  management  functions  and  multicast 
routing  outside  of  Interne!  gateways,  in  special  hosts  called 
multicast  ay.ents.  The  collection  of  multicast  agents  in  effect 
provides  a  second  gateway  system  on  lop  of  the  existing  Internet, 
for  muliirasl  purposes.  Hie  major  costs  of  Ihis  separation  arc 
redundancy  of  routing  tables  tx'lwccn  gateways  and  multicast 
agents  and  (he  increased  delay  and  unreliability  of  extra  hops  in 


the  delivery  path.  Much  of  the  touting  information  in  the 
multicast  agents  must  be  "wired-in"  because  they  do  not  have 
access  to  the  gateways'  routing  tables.  I  lowcvcr.  this  rudimentary 
implementation  provides  an  environment  for  evaluating  the 
interface  to  the  multicast  service  and  for  investigating  group 
management  and  multicast  routing  protocols  for  eventual  use  in 
the  gateways.  It  also  serves  as  a  testbed  for  porting  multicast- 
based  distributed  applications  to  an  internetwork  from  the  V 
distributed  operating  system. 

For  now,  we  arc  restricting  group  membership  to  local 
networks  that  already  have  a  broadcast  or  multicast  capability, 
such  as  the  lilhemet  We  feel  that,  in  the  future,  any  network  that 
is  to  support  hosts  other  than  just  gateways  must  have  a  multicast 
addressing  mode.  Ffficient  implementation  of  multicast  within 
point-to-point  or  virtual  circuit  networks  deserves  investigation. 

A  significant  issue  raised  by  the  host  group  model  is 
authentication  and  access  control  in  internetworks.  Gateways 
must  control  which  hosts  can  create  and  join  host  groups, 
presumably  making  their  decision  based  on  the  identity  of  the 
requestor  (thus  requiring  authentication)  and  permissions  (access 
control  lists).  This  issue  docs  not  arise  in  conventional 
internetwork  architectures  because  host  addresses  are 
administratively  assigned  with  no  notion  of  dynamic  assignment 
and  binding  as  provided  by  host  groups.  We  believe  that  access 
control  should  be  recognized  as  a  proper  and  necessary  function 
of  gateways  so  as  to  protect  the  hosts  of  local  networks  from 
general  internetwork  activity.  Thus,  group  access  control  can  be 
subsumed  as  part  of  this  more  general  mechanism,  although  more 
investigation  of  the  general  issue  is  called  for. 

On  a  philosophical  point,  there  has  been  considerable 
reluctance  to  make  open  use  of  multicast  on  local  networks 
because  it  was  network -specific  and  not  provided  across 
internetworks.  We  were  originally  of  that  school.  However,  we 
recognized  that  our  'hidden*  uses  of  multicast  in  the  V 
distributed  system  were  essential  unless  we  resorted  to 
dramatically  poorer  solutions  -  wired-in  addresses.  We  also 
recognized,  as  described  in  this  paper,  that  an  adequate  multicast 
facility  for  internetworks  was  feasible.  As  a  consequence,  we  now 
argue  that  multicast  is  an  important  and  basic  facility  to  provide 
in  local  networks  and  internetworks.  Higher  levels  of 
communication,  including  applications,  should  feel  free  to  make 
use  of  this  powerful  facility.  Networks  and  internetworks  lacking 
multicast  should  be  regarded  as  deficient  relative  to  (he  future 
(and  present)  requirements  of  sophisticated  distributed 
application-sand  communication  systems. 
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